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1.0  SUMMARY 


The  complexity  of  modem  defense  systems  is  growing  constantly.  New  technologies 
create  opportunities  for  higher  levels  of  integration.  Modem  systems  such  as  air  and  ground 
vehicles  contain  a  larger  number  of  components  that  interact  with  each  other  in  non-linear  and 
often  unpredictable  ways.  Unintended  interactions  lead  to  unexpected  behaviors  and 
consequences,  some  of  which  have  proven  to  be  catastrophic.  A  key  technical  challenge  in 
developing  such  complex  systems  is  to  ensure  that  catastrophic  subsystem  and  component 
interactions  are  well  understood  and  contained  prior  to  full-scale  development. 

To  address  these  challenges,  The  Defense  Advanced  Research  Projects  Agency 
(DARPA)  is  investing  in  novel  methods  for  design  and  verification  of  complex  systems.  The 
META  program  (META  not  an  acronym  but  is  typically  spelled  using  all  capital  letters  by 
DARPA)  is  specifically  aimed  at  compressing  the  product  development  and  deployment  timeline 
of  complex  defense  systems  through  model-based  design  and  manufacturing.  Using  the  META 
design  paradigm,  different  component  model  libraries  can  be  used  to  instantiate,  analyze,  and 
verify  a  system  design  independent  of  its  physical  manifestation.  The  goal  is  to  establish  a 
“correct-by-construction”  design  prior  to  detailed  design  and  prototyping. 

Under  the  META  program,  a  team  led  by  the  PARC  (Palo  Alto  Research  Center)  team  is 
developing  a  model-based  system-engineering  framework  that  enables  architectural  analysis  of 
complex  systems  during  the  conceptual  design  phase.  Using  this  framework,  design  teams  can 
systematically  explore  architectural  design  decisions  during  the  early  stage  of  system 
development  prior  to  the  selection  of  specific  components.  The  analysis  performed  at  this  earliest 
stage  of  design  facilitates  the  development  of  more  robust  and  reliable  system  architectures.  This 
report  provides  a  summary  of  the  work  conducted  by  the  PARC  team  during  the  course  of  the 
one -year  project. 
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2.0  INTRODUCTION 


The  complexity  of  modem  defense  systems  is  growing  constantly.  New  technologies 
create  opportunities  for  higher  levels  of  integration.  Modem  systems  such  as  air  and  ground 
vehicles  contain  a  larger  number  of  components  that  interact  with  each  other  in  non-linear  and 
often  unpredictable  ways.  Unintended  interactions  lead  to  unexpected  behaviors  and 
consequences,  some  of  which  have  proven  to  be  catastrophic.  A  key  technical  challenge  in 
developing  such  complex  systems  is  to  ensure  that  catastrophic  subsystem  and  component 
interactions  are  well  understood  and  contained  prior  to  full-scale  development. 

To  address  these  challenges,  DARPA  is  investing  in  novel  methods  for  design  and 
verification  of  complex  systems.  The  META  program  is  specifically  aimed  at  compressing  the 
product  development  and  deployment  timeline  of  complex  defense  systems  through  model-based 
design  and  manufacturing.  Using  the  META  design  paradigm,  different  component  model 
libraries  can  be  used  to  instantiate,  analyze,  and  verily  a  system  design  independent  of  its 
physical  manifestation.  The  goal  is  to  establish  a  “correct-by-construction”  design  prior  to 
detailed  design  and  prototyping. 

Eor  mission-critical  design  applications,  a  key  consideration  is  the  ability  of  the  designed 
system  to  meet  specified  requirements.  Ideally,  designers  aim  to  converge  on  an  architecture 
with  a  high  probability  of  meeting  design  requirements  as  early  as  possible  during  the  design 
phase.  Availability  of  high-fidelity  models,  simulation  frameworks,  and  other  analytical  tools 
greatly  simplifies  this  process.  Arguably,  design  automation  is  well  advanced  in  certain  other 
fields  such  as  integrated  circuit  (IC)  design.  Whether  concepts  and  methods  from  IC  design 
could  be  adapted  to  the  design  of  electromechanical  systems  is  a  subject  of  much  debate  in  the 
design  community  [1,2].  A  recent  approach  named  Platform-Based  Design  aims  to  increase  the 
level  of  automation  in  the  design  of  complex  electromechanical  systems  by  introducing  multiple 
levels  of  abstraction  [2].  Nevertheless,  design  complexity  remains  a  significant  challenge  in  the 
design  of  electromechanical  systems,  often  resulting  in  substantial  delays  and  cost  overruns  in 
the  design  of  military  and  aerospace  systems. 

Under  the  META  program,  a  team  led  by  PARC  is  developing  a  model-based  system¬ 
engineering  framework  that  enables  architectural  analysis  of  complex  systems  during  the 
conceptual  design  phase.  Using  this  framework,  design  teams  can  systematically  explore 
architectural  design  decisions  during  the  early  stage  of  system  development  prior  to  the  selection 
of  specific  components.  The  analysis  performed  at  this  earliest  stage  of  design  facilitates  the 
development  of  more  robust  and  reliable  system  architectures.  This  report  provides  a  summary  of 
the  work  conducted  by  the  PARC  team  during  the  course  of  the  one-year  project. 

One  quantitative  goal  of  the  META  program  is  to  compress  the  system  design, 
development,  test,  and  evaluation  (DDTE)  cycle  by  a  factor  of  5x  or  more.  The  PARC-led 
META-II  project  supports  this  goal  by:  1)  identifying  design  defects  early  in  the  design  process 
using  design-stage  models  and  abstractions;  and  2)  reducing  the  time  and  effort  required  for 

2 


Approved  for  public  release;  distribution  unlimited. 


system  verification  through  hardware-in- the-loop  testing  using  co-verification  of  hardware  and 
software  at  the  design  stage.  Design-stage  analyses  help  identify  system-level  and  component 
interaction  problems  early  in  the  DDTE  cycle  and  thus  prevent  costly  redesigns  during  later 
design  or  deployment  phases. 


2,1  Impact  Statement 

We  believe  that  a  systematic  approach  that  identifies  design  defects,  uncertainty,  and 
unforeseen  fault  propagation  effects  during  the  early  design  process  will  compress  DDTE 
schedules  significantly.  Desired  improvements  of  >5x  are  within  reach  if  the  methods 
developed  under  the  META  program  are  widely  adopted  in  the  defense  community.  Much  of  the 
complexity  and  cost  of  verification  is  mandated  through  complex  policies  such  DO-254,  DO- 
178B,  and  NASA’s  NPR  8705. 2B  (Human  Rating  Requirements).  We  envision  that  a  successful 
conclusion  to  these  programs  would  yield  revolutionary  new  methods,  tools,  and  processes  that 
will  help  refine  the  rigid  procedural  requirements  that  impose  substantial  delays  and  cost 
overruns  on  every  major  aerospace  program  today. 

During  this  project,  a  stated  goal  of  the  PARC  team  has  been  to  generate  and  verify 
robust  conceptual  designs  in  an  automated  fashion  in  far  less  time  than  the  manual  system 
engineering  approach.  The  impact  of  this  achievement  will  be  early  elimination  of  design  flaws, 
reduced  need  for  hardware-in-the-loop  testing,  and  compressed  design  cycle  times. 
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3.0  METHODS,  ASSUMPTIONS,  AND  PROCEDURES 


3.1  The  META-II  Tool  Chain  and  Design  Workflow 

Design  of  complex  systems  for  mission-critical  applications  is  driven  by  the  ability  of  the 
system  to  meet  specified  requirements.  Typically,  the  performance  of  the  system  with  respect  to 
requirements  is  verified  using  costly  and  time-consuming  tests  of  prototype  hardware  under 
realistic  operational  scenarios.  Failure  to  verify  performance  requirements  at  this  stage 
necessitates  additional  design/prototyping/test  cycles  and  often  results  in  cost  and  schedule 
overruns. 

The  PARC  META  effort  is  guided  by  three  key  insights  about  the  traditional  design 
process: 

1 .  Traditional  system  design  activities  explore  a  very  limited  range  of  the  overall 
design  space,  often  informed  by  what  has  worked  well  in  the  past  and  what  did 
not. 

2.  In  traditional  system  design,  detailed  analysis  of  functional  failures  and  fault 
propagation  is  performed  late  during  the  design  process,  long  after  the  design 
team  has  -  figuratively  speaking  -  painted  themselves  into  a  corner. 

3.  There  is  ample  information  about  component  and  system  behavior  and  function 
available  during  the  conceptual  design  process  to  initiate  system  verification. 

Based  on  these  key  insights,  the  PARC-led  team  is  developing  an  integrated  design  flow 
and  verification  tool  chain.  The  PARC  tool  chain  uses  a  component  model  library,  design  rules 
for  the  target  domain,  and  system  requirements  as  input.  Using  these  inputs,  the  PARC  tool  chain 
automates  the  generation,  evaluation,  and  verification  of  conceptual  system  models  in  a  seven- 
step  process: 

1 .  A  large  number  of  candidate  functional  designs  are  generated  using 
configurational  requirements  for  the  target  system,  a  component  model  library, 
design  rules,  and  a  generative  grammar.  The  design  synthesis  step  ensures  that  the 
design  space  is  adequately  explored  and  no  novel  architectural  solutions  are  left 
on  the  table  early  on  in  the  process. 

2.  Candidate  designs  are  evaluated  with  respect  to  robustness  to  known  fault  modes 
using  a  functional  simulation.  This  step  is  the  functional  equivalent  of  a  Failure 
Modes  and  Effects  Analysis  (FMEA)  and  it  ensures  that  only  the  most  robust 
designs  are  promoted  to  the  final  step. 

3.  Next,  we  use  a  set  of  system-level  metrics  (developed  by  other  META  performer 
teams)  to  select  the  strongest  candidate  designs.  Applicable  metrics  include 
adaptability,  complexity,  and  flexibility. 

4.  Selected  candidate  designs  are  verified  using  a  functional  verification  process 
based  on  state-of-the-art  model  checking  technology.  Note  that  the  functional 
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verification  process  may  be  applied  to  functions  embodied  in  software  as  well  as 
hardware. 

5.  A  performance  verification  process  utilizes  detailed  models  of  component 
behavior  in  order  to  compute  the  probability  of  correctness  (PoC)  of  each 
candidate  design  with  respect  to  a  set  of  performance  requirements. 

6.  A  traditional  reliability  analysis  is  performed  on  the  most  promising  candidate 
designs. 

7.  Information  obtained  from  functional  verification,  performance  verification, 
FMEA,  and  reliability  analysis  tasks  are  composed  into  a  Probabilistic  Certificate 
of  Correctness  (PCC)  for  the  candidate  designs. 


Figure  1  illustrates  the  overall  process  and  the  major  elements  of  the  automated  tool  chain 
for  model-based  system  engineering. 


Figure  1 .  The  Model-Based  System-Engineering  Process  Flow 
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3,2  Component  Library 


A  key  ingredient  of  an  automated  tool  chain  for  model-based  system  engineering  is  a 
component  model  library  that  is  used  to  generate  candidate  designs  for  the  target  system.  The 
component  model  library  is  question  consists  of  functional  and  behavioral  models,  interfaces, 
and  ranges  of  input,  output,  and  state  variables  for  representative  components  for  the  electrical 
power  system  (EPS)  domain  (of  any  other  domain  of  interest).  We  chose  the  Modelica  language 
to  develop  the  component  libraries  for  the  META  tool  chain'.  The  EPS  library  provides 
Modelica  models  for  the  behavior  and  function  of  EPS  components  such  as  batteries,  actuators, 
electrical  switches,  etc.  Eor  each  component,  the  nominal  behavior  was  modeled  and  augmented 
with  the  relevant  failure  modes.  The  nominal  and  the  fault  mode  behavior  of  the  EPS  Eibrary 
components  were  validated  by  comparing  the  simulated  behavior  of  test  models  with 
measurements  and  sensor  data  from  the  Advanced  Diagnostics  and  Prognostics  Testbed 
(ADAPT)  at  NASA  Ames  Research  Center. 

Eor  several  components,  we  created  models  with  different  levels  of  accuracy.  Eor 
example,  the  inverter  component  models  range  from  very  simple  models  that  describe  only  the 
AC/DC  power  balance  equation  to  models  containing  complicated  electrical  schematics 
including  semiconductor  components  from  the  standard  Modelica  library  for  electrical  systems. 
The  reason  for  creating  models  of  the  same  component  with  different  levels  of  detail  is  to 
compare  the  performance  of  our  tool  chain  during  early  stages  of  conceptual  analysis  versus  later 
stages  when  component  selection  is  almost  final  and  more  details  are  available  regarding 
component  performance  and  interactions.  A  recent  article  describes  the  EPS  component  model 
library  in  greater  detail  [3]. 


3,3  Design  Space  Exploration 

The  first  step  in  the  automated  tool  chain  is  to  explore  the  space  of  feasible  conceptual 
designs.  As  mentioned  earlier,  traditional  system  design  activities  explore  a  very  limited  range  of 
the  overall  design  space.  Successful  past  designs  often  provide  templates  on  which  new  designs 
are  based.  Consequently,  many  innovative  yet  unorthodox  design  alternatives  often  go 
unexplored.  As  an  illustration,  consider  the  variety  and  diversity  of  flying  machine  designs 
introduced  during  the  first  couple  of  decades  of  aviation.  Over  time,  two  templates  (fixed  wing 
with  horizontal  and  vertical  stabilizers,  and  rotary  wing  with  a  tail  rotor)  gained  momentum  and 
suppressed  the  exploration  of  alternatives  for  most  new  aircraft  design  projects.  On  occasion,  we 


'  Modelica  Association.  (2011).  Modelica  and  the  Modelica  Association.  Available  at 
https://http://www.mode  lica.org/. 
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see  examples  of  innovative  design  appear  in  eertain  high-performance  aircraft  (e.g.,  canards, 
pusher  props).  A  design  space  exploration  process  allows  designers  to  explore  the  vast  space  of 
potential  designs  that  meet  configurational  requirements  for  the  system  in  a  systematic  fashion. 

Recently,  engineering  design  researchers  have  discovered  that  graph  grammars  provide  a 
flexible  yet  ideally  structured  approach  to  the  creation  of  complex  engineering  systems  [4].  This 
interpretation  of  the  design  process  makes  graph  grammars  very  suitable  for  computationally 
modeling  the  open-ended  nature  of  conceptual  design,  where  designers  explore  various  ideas, 
decisions,  and  modifications  to  previous  de-  signs  to  arrive  at  feasible  solutions. 

In  our  approach,  we  use  a  graph-grammar-based  design  space  exploration  technique 
[3,4].  This  generative  technique  takes  user  specified  EPS  loads  as  input  and  satisfies  system- 
level  configuration  requirements  to  generate  feasible  EPS  candidate  architectures.  Eor  instance, 
if  there  were  configurational  requirements  to  provide  reliable  power  to  certain  electrical  loads, 
our  approach  would  generate  many  feasible  designs  with  alternative  configurations  for  power 
generation  (redundant  and  dissimilar  sources),  distribution  (e.g.,  redundant  buses),  and 
switching. 

A  component  model  library  serves  as  the  backbone  of  our  design  space  exploration 
approach  [3].  Eising  the  models  in  the  model  library  as  building  blocks,  this  generative  graph 
grammar  based  technique  configures  “correct  by  construction”  EPS  architectures  that  can  be 
further  studied  by  means  of  a  simulation-based  analysis.  The  graph-grammar-based 
configuration  approach  uses  graphs  as  a  representation  scheme.  This  approach  captures  the 
transitions  or  the  production  rules  for  creating  a  solution,  as  opposed  to  storing  the  solutions 
themselves.  The  initial  specification  is  represented  as  a  simple  graph  in  which  the  desired  inputs 
and  outputs  are  cast  as  arcs  and  nodes  of  the  to-be-designed  artifact.  The  evolution  of  a  design 
from  its  inception  to  its  final  configuration  can  be  viewed  as  a  progression  of  graph 
transformations  that  lead  to  the  final  configuration. 

Generating  feasible  EPS  architectures  using  graph-grammar-based  configuration  is  a  two- 
step  process.  In  the  first  step,  we  provide  an  EPS  system  design  grammar  to  encode  design  rules 
for  constructing  EPS  architectures.  The  rules  are  established  prior  to  the  design  process.  We  have 
found  that  a  14-rule  graph  grammar  is  sufficient  to  generate  feasible  EPS  architectures  from 
multiple  EPS  requirements  [3].  As  an  example,  a  prototypical  EPS  design  requirement  is  to 
provide  a  battery  relay  for  each  battery  circuit  in  order  for  the  flight  crew  to  isolate  the  battery 
from  the  rest  of  the  EPS  in  case  of  a  malfunction.  Corresponding  design  rules  govern  the 
mapping  of  functional  requirements  to  components,  or  the  physical  compatibility  between  EPS 
components.  Eor  instance,  a  design  rule  for  the  EPS  domain  would  insert  a  battery  relay  for  each 
circuit  where  a  battery  is  present.  Specifically,  the  design  grammar  encodes  how  specific  system 
requirements  can  be  embodied  by  selecting  components  from  a  full  spectrum  of 
electromechanical  components  represented  in  the  component  library. 
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In  the  second  step,  the  graph  transformation  systems,  or  graph  grammars,  is  invoked 
algebraically.  Algebraic  graph  transformation  methods  rigorously  define  mathematical 
operations  such  as  addition  and  intersection  of  graphs.  A  typical  graph  grammar  rule  is 
comprised  of  a  left-hand  side  and  a  right-hand  side  (Figure  2).  The  LHS  contains  the 
preconditions  that  trigger  the  rule.  Once  the  rule  is  triggered,  the  RHS  describes  the  resulting 
graph  transformation.  By  simply  executing  different  combinations  of  grammar  rules,  a  variety  of 
feasible  EPS  architectures  can  easily  be  generated.  Using  rules  to  support  component  redundancy 
in  the  architectures,  we  are  able  to  generate  thousands  of  correct  models  using  multiple  starting 
seeds  in  a  matter  of  minutes.  Figure  3  illustrates  a  candidate  design  generated  using  a  single  seed 
and  a  graph  grammar  of  14  EPS-related  rules. 


Figure  2.  An  Example  Graph  Grammar  Rule  for  an  EPS 


Figure  3.  A  Design  Concept  Generated  for  the  EPS  Domain 
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3.4 


Functional  Failure  Analysis 


Functional  Failure  Analysis  (FFA)  is  a  recent  approach  for  coupling  failure  analysis  with 
product  design  during  the  conceptual  stage  [5].  The  FFA  approach  is  based  on  the  notion  that  a 
failure  happens  when  a  functional  element  in  the  system  does  not  perform  its  intended  task.  For 
eaeh  funetion  that  is  subjeet  to  failure,  a  funetional  eritieality  is  defined  depending  on  the  role  of 
funetionality  in  aceomplishing  designed  tasks.  A  simulation-based  failure  analysis  tool  is  then 
used  to  analyze  functional  failures  and  reason  about  their  impact  on  overall  system  functionality. 
Using  this  framework,  design  teams  can  systematically  explore  risks  and  vulnerabilities  during 
the  early  (functional  design)  stage  of  system  development  prior  to  the  seleetion  of  speeific 
eomponents.  In  essenee,  FFA  provides  early  insight  into  the  robustness  of  a  design  against 
known  fault  modes,  much  like  the  FMEA  proeess  that  is  often  performed  at  the  end  of  the  design 
cycle. 


Using  FFA,  a  multitude  of  failure  scenarios  can  be  quickly  analyzed  to  determine  the 
effects  of  architectural  design  deeisions  on  overall  system  functionality.  In  the  META  tool  chain, 
each  candidate  generated  during  the  first  step  is  further  analyzed  using  EEA.  The  tool  ehain 
automatieally  generates  single,  multiple,  and  easeading  fault  seenarios  and  measures  the  impact 
of  each  fault  on  overall  system  function.  Eigure  4  illustrates  the  result  of  a  single  simulation  run 
over  a  candidate  EPS  design.  Once  the  entire  suite  of  functional  loss  scenarios  is  simulated,  the 
tool  ehain  generates  an  overall  seore  for  the  candidate  design.  This  seore  is  eorrelated  with  the 
robustness  of  the  candidate  design  with  respect  to  known  failure  modes. 


^226^J^240^J^242 ^ 


• 

• 

Battery 

2 

m 

-j  CB236  ^ 

■|  EY244  H 

CB:  Circuit  Breaker 
E:  Voltage  Sensor 
ESH:  Position  Sensor 
EY:  Relay 

FT:  PressureSensor 
ISH:  Position  Sensor 
IT:  Current  Sensor 
ST:  RPM  Sensor 
TE:  Temperature  Sensor 


^ CB280  00465  \ 


.^0 


ESH 

244 


l28lii28li 


Functional 
Health  Status 

#  Operating 

#  Degraded 

0  Lost  •  Recoverable 

#  Lost 


Eigure  4.  Results  of  an  EEA  Eault  Propagation  Simulation  on  an  EPS  Design 
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3.5 


Functional  Verification 


The  next  step  in  the  META  tool  chain  is  the  functional  verification  of  selected  candidate 
designs.  We  interpret  functional  verification  as  a  probabilistic  evaluation  of  a  composite 
functional  model  of  the  system  against  functional  requirements  imposed  on  the  system.  In  order 
to  accomplish  functional  verification,  the  first  step  is  to  compose  a  functional  model  of  the 
candidate  system  by  retrieving  functional  models  for  each  component  from  the  component  model 
library  and  connecting  those  model  fragments  following  the  topology  prescribed  in  the  candidate 
design.  Next,  we  use  a  probabilistic  model  checker  called  PRISMATIC  to  perform  functional 
verification  on  these  models.  PRISMATIC  is  based  on  PRISM,  an  open  source  tool  for  formal 
modeling  and  analysis  of  systems  that  exhibit  random  or  probabilistic  behavior  [6].  PRISMATIC 
supports  a  wide  range  of  probabilistic  models  including  discrete-time  and  continuous-time 
Markov  chains,  Markov  decision  processes,  probabilistic  automata,  probabilistic  timed  automata, 
plus  extensions  of  these  models  with  costs  and  rewards.  Models  are  described  using  a  simple, 
state-based  language.  PRISMATIC  provides  support  for  automated  analysis  of  a  wide  range  of 
quantitative  properties  of  these  models,  including: 

•  Internal  model  consistency  (e.g.,  only  one  failure  mode  is  active  for  any  component  at  any 
time); 

•  Fault  probability  analysis  (e.g.,  what  is  the  probability  that  some  fault  occurs  up  to  time  T,  or 
steady  state); 

•  Time  bounded  functional  assessment  (e.g.,  what  is  the  probability  that  some  component  is 
functioning  normally  at,  before,  or  up  to  time  T); 

•  Limited-fault  analysis  (e.g.,  if  exactly  X  component(s)  fail(s)  by  time  T  (or  steady  state, 

“long  run”),  what  is  the  effect  on  other  failure  modes  or  functions?). 


3,6  Performance  Verification 

Another  important  aspect  of  certification  is  the  verification  of  a  system  (or  a  system 
design)  with  respect  to  performance  requirements.  In  later  stages  of  development,  such 
performance  verification  is  often  performed  through  hardware-in-the-loop  testing.  During  the 
early  (conceptual)  design  stage,  performance  verification  has  to  be  accomplished  using  models 
of  system  behavior.  However,  two  types  of  uncertainty  confound  the  use  of  models  for 
verification.  Epistemic  uncertainty  comes  from  model  fidelity  and  inaccuracy,  and  aleatory 
uncertainty  is  contributed  by  inherent  limitations  in  estimating  relevant  system  parameters  and 
environmental  variables.  Thus,  model-based  performance  verification  has  to  take  uncertainty  into 
account.  To  address  these  needs,  we  employ  uncertainty  propagation  (UP)  methods  used  in 
reliability-based  design  [7]  to  quantify  the  PoC  of  a  given  system  design. 

Common  UP  methods  can  be  classified  in  five  broad  categories: 

•  Simulation-based  methods  such  as  Monte  Carlo  simulation; 
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•  Local  expansion-based  methods  like  the  Taylor  series  method  or  perturbation  method; 

•  Most  probable  point-based  methods  such  as  the  first-order  reliability  method  and  seeond- 
order  reliability  methods; 

•  Funetional  expansion-based  methods,  sueh  as  the  Neumann  expansion  and  the  polynomial 
ehaos  expansion; 

•  Numerieal  integration-based  methods,  where  the  statistieal  moments  are  first  ealeulated  by 
direet  numerieal  integration,  and  then  the  probability  density  or  the  tail  region  probability  is 
approximated  using  an  empirieal  distribution  system  based  on  the  ealeulated  moments. 

A  detailed  deseription  of  these  methods  and  their  applieation  in  PoC  eomputation  is 
beyond  the  scope  of  this  paper.  Further  details  of  our  approach  to  UP  may  be  found  in  a  reeent 
publication  by  our  team  [8].  Most  importantly,  none  of  these  methods  is  elearly  superior  to 
others.  Depending  on  the  type  of  models  available  (qualitative,  quantitative,  or  hybrid),  funetion 
types  (linear,  quadratie,  highly  non-linear),  input  distributions  (parametric  or  non-parametric), 
and  input/output  distribution  types  (normal,  uniform,  beta,  or  exponential),  we  have  devised  an 
algorithm  to  choose  the  most  suitable  UP  method  from  a  library  of  six  different  methods. 

The  first  step  in  the  performance  verifieation  proeess  is  to  eompute  the  PoC  of  the  model 
(or  a  model  fragment)  with  respeet  to  a  single  performanee  requirement.  For  instanee,  we 
eomputed  the  PoC  for  an  EPS  subsystem  eontaining  a  battery,  an  AC  inverter,  and  three  loads 
eonsisting  of  an  eleotrieal  motor,  a  pump,  and  a  light.  The  left  hand  graphie  on  Figure  5 
illustrates  the  result  of  PoC  eomputation  for  this  system  with  respect  to  a  requirement  that  states 
that  the  motor  speed  shall  remain  between  900  and  1000  RPM  during  nominal  operation.  The 
right  hand  side  of  the  same  figure  illustrates  the  result  of  a  seeond  PoC  eomputation  for  the  same 
system  with  respeet  to  a  different  requirement  that  requires  the  pump  voltage  to  remain  between 
117  and  125  VAC.  In  this  example,  the  PoC  with  respeet  to  the  first  requirement  is  eomputed  as 
0.881  whereas  the  PoC  for  the  seeond  requirement  is  0.899.  The  solid  red  areas  indicate  the  total 
probability  mass  assoeiated  with  meeting  eaeh  requirement. 
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Figure  5.  PoC  Computation  for  an  EPS  Model  with  Respect  to  Two  Different  Requirements 


The  second  step  in  the  performance  verification  process  is  to  compute  the  joint 
probability  of  meeting  multiple  requirements.  Note  that  the  joint  probability  cannot  be  obtained 
by  simply  multiplying  the  two  marginal  probabilities.  Instead,  we  need  to  compute  the 
covariance  matrix  for  the  marginal  probabilities. 

Z  =  cov{Yi,Yj)  =  E[{Yi-mYj-Yj)]^^^ 

For  the  same  example  above,  the  covariance  matrix  calculation  yields  a  joint  probability 
of  0.838  for  meeting  both  requirements  at  the  same  time  (note  that  this  PoC  is  higher  than  the 
simple  multiplication  of  the  two  individual  PoCs,  which  would  have  yielded  0.792). 
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As  a  final  caution  regarding  the  joint  probability  computation,  we  cannot  use  the 
covariance  matrix  computation  if  the  marginal  distributions  are  different.  In  that  case,  we  use  the 
Gaussian  Copula  function  to  approximate  a  true  multivariate  distribution.  Using  the  Copula 
function,  we  can  join  different  types  of  distributions  (normal  and  beta,  etc.). 

Cy{ii\.ii2 . tiM )  =  4'Ar(0.i;)  i )  ( “ ' ) . i )  ( ) ) 

=  (2) 


3,7  PoC  and  PCC  Computation 

We  define  correctness  as  a  measurement  of  how  well  the  system  design  meets  or  exceeds 
its  requirements.  Due  to  uncertainty  arising  from  design  abstractions  as  well  as  system 
interactions  with  the  environment  and  its  users,  correctness  is  specified  as  a  probability 
distribution  over  multiple  dimensions.  These  dimensions  include  the  design  hierarchy  (i.e., 
subsystems  and  components),  environment,  use  conditions,  functions,  functional  failures,  and 
adherence  to  design  rules. 

We  define  the  Probabilistic  Certificate  of  Correctness  (PCC)  as  a  data  structure  that 
documents  the  correctness  of  a  particular  system  design  with  respect  to  each  class  of  pertinent 
system  requirements.  At  the  system  level,  the  PCC  is  represented  as  a  multidimensional 
probability  density  function  where  each  dimension  represents  a  distinct  system  requirement.  The 
PCC  is  further  organized  in  a  hierarchy  that  provides  PCCs  at  subsystem  and  component  levels. 
Given  a  particular  use  case  (represented  as  scalar  values  of  performance  metrics  mentioned  in  the 
requirements),  the  PCC  provides  a  probability  of  correctness  for  the  design  under  those  use 
conditions.  This  approach  allows  the  design  and  development  process  to  focus  on  areas  of  low 
probability  of  correctness  for  further  refinement  and  elaboration. 

The  PoC  is  represented  as  a  probability  distribution  (or  probability  density  function) 
reflecting  the  probability  that  the  given  design  meets  a  performance  requirement  or  a  set  of 
related  requirements.  It  is  possible  to  combine  the  PoCs  into  a  unified  PCC.  Since  the  behavior 
of  models  is  an  approximation  of  actual  system  performance,  the  PCC  process  takes  this 
uncertainty  into  account  and  provides  information  on  the  resulting  uncertainty  as  well  as  what 
components  contribute  to  the  uncertainty.  The  resulting  PCC  serves  as  the  formal  record  of 
performance  verification  record  for  the  candidate  design.  Most  importantly,  unlike  traditional 
design  flow,  this  performance  verification  is  achieved  using  only  (virtual)  models  of  the  system. 

Our  tool  chain  and  framework  is  the  basis  for  specifying  system  requirements,  supporting 
design  space  exploration,  and  analyzing  the  performance  associated  with  promising  architectural 
design  alternatives.  To  support  a  model-based  design  paradigm,  the  framework  allows  the 
designers  to  combine  models  from  different  domains  into  integrated  system  level  models,  and 
allow  models  of  components  and  sub-systems  to  evolve  throughout  the  design  process.  At  the 
end,  component  models  are  composed  into  a  system  that  achieves  the  intended  functionality 
given  specified  requirements  such  as  reliability,  risk,  and  performance. 
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3.8  Reliability  Analysis 


The  final  analytical  operation  in  the  tool  chain  is  a  traditional  reliability  analysis.  Our 
method  is  based  on  identification  of  criticality  and  sensitivity  of  system  components,  and  a 
simulation  model  that  incorporates  probability  and  failure  rates  of  individual  components  such 
that  system-level  reliability  measures  can  be  computed.  Computed  reliability  measures  include 
expected  system  lifetime,  component  criticality  and  sensitivity  values,  and  weighted  or 
maximum  failure  probability  [9].  Future  work  may  involve  integration  of  computed  criticality 
data  with  the  FFA  method  in  order  to  generate  the  equivalent  of  a  Failure  Modes,  Effects,  and 
Criticality  Analysis  (FMECA)  during  the  conceptual  design  phase. 

3,9  System  Integration 

The  Meta-II  software  tool  chain  is  a  heterogeneous  set  of  software  systems  developed  in 
five  programming  and  modeling  languages  by  four  organizations  across  the  United  States.  It 
includes  significant  legacy  code  bases  and  newly  developed  reasoning  and  integration  code. 
PARC  has  leveraged  its  location  within  Silicon  Valley  to  apply  state-of-the  industry  software 
engineering  practices  to  ensure  successful  software  development  and  integration. 

The  PARC  team  followed  a  best-practice  agile  software  development  process. 
Development  was  planned  in  short  two-week  iterations  that  emphasize  the  frequent  delivery  of 
working  software  that  demonstrates  progress  allowing  us  to  assess  if  the  project  is  moving  in  the 
right  scientific  direction. 

The  team  coordinated  its  work  through  the  JIRA  issue-tracking  system  available  to  all 
members  through  a  Web  based  interface,  which  provides  visibility  on  task  state  and  is  especially 
effective  at  identifying  dependencies  between  team  members. 

Software  source  was  controlled  via  a  subversion  repository  and  change  notifications  are 
emailed  immediately  to  team  members.  This  practice  ensures  the  team  is  kept  up  to  date  and  also 
creates  peer  pressure  to  write  clean  code,  as  it  is  visible  to  all. 

PARC  has  taken  a  test-driven  approach  to  new  software  development  by  providing  unit 
tests  and  monitoring  our  cost  test  coverage  levels.  On  each  code  check-in,  the  entire  system  is 
built  and  run  together  with  unit  and  integration  tests  using  a  Hudson  continuous  build  server. 

PARC  has  used  industry  standard  tools,  techniques,  and  design  patterns  in  this  project. 
META-II  is  built  using  the  ANT  build  system.  Tests  are  written  using  the  JUnit  and  Mockito 
frameworks.  Code  coverage  is  monitored  using  the  Clover  test  coverage  tool.  Eogging  is 
controlled  through  JEog. 
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The  software  engineers  at  PARC  and  MCT  partieipate  in  weekly  eode  review  meetings 
with  all  the  engineers  in  the  Embedded  Reasoning  Area  of  PARC  team.  This  group  with  broad 
researeh  and  industry  experience  ensures  that  the  team  keep  up  to  date  on  industry  practice  and 
provides  a  forum  for  peer-reviewing  software  architecture  and  design  issues. 
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4.0  RESULTS  AND  DISCUSSIONS 


The  PARC  team  has  used  two  case  studies  to  design,  develop,  and  test  the  META-II  tool 
chain  and  to  demonstrate  the  features  of  the  proposed  verification  process.  These  are  the 
ADAPT  EPS  testbed  at  NASA  Ames  and  the  Ramp  System  of  an  Infantry  Fighting  Vehicle 
(IFV).  The  two  systems  have  been  modeled  with  the  help  of  the  Modelica  language.  Because 
Modelica  is  a  system  modeling  language  as  opposed  to  a  programming  language,  Matlab  is  used 
to  code  the  various  UP  methods.  Matlab  calls  the  black  box  OpenModelica  model  using  the 
OpenModelica-Matlab-Interface  tooP,  requiring  only  that  the  input  and  output  variables  be 
known  from  the  Modelica  model,  but  not  the  functional  relationships  coded  in  Modelica. 

The  following  subsections  describe  in  details  the  two  study  systems,  the  associated 
models  and  the  experiments  performed  on  the  models. 

4.1  ADAPT  Electrical  Power  System  Example 

4.1.1  Domain  Description 

ADAPT,  pictured  in  Figure  6,  consists  of  a  controlled  and  monitored  environment  where  faults 
are  injected  into  the  system  in  a  controlled  manner  and  the  performance  of  the  test  article  is 
carefully  monitored.  The  hardware  of  the  testbed  represents  an  EPS  for  a  hypothetical  space 
exploration  vehicle. 


^  Schaad,  C.,  2009.  OpenModelica  Matlab  functions. 
http://openmodelica.ida.liu.se:8080/cb/proi/doc.do?doc  id=1085. 
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Figure  6.  The  Advanced  Diagnostics  and  Prognostics  Testbed  (Courtesy  of  NASA  Ames 

Research  Center) 


The  ADAPT  system  consists  of  three  major  modules:  a  power  generation  unit,  a  power  storage 
unit  and  a  power  distribution  unit.  The  interconnections  among  the  different  units  are  depicted  in 
Figure  7. 


Power  Generation 


Power  Storage 


Power  Distribution 


Figure  7.  The  ADAPT  Testbed  Structure  (Picture  Reproduced  from  [10]) 
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The  Power  Generation  unit  can  charge  the  batteries  located  in  the  Power  Storage  Unit 
with  the  help  of  two  battery  chargers  and  a  photovoltaic  unit  (solar  panel).  The  power  generation 
unit  is  divided  into  six  subsystems;  the  solar  panel  unit,  the  battery  charger  panel,  the  protection 
and  enable  panel  and  three  battery-charge  selection  panels.  The  power  storage  unit  contains  three 
battery  packs  and  several  relays  that  control  the  connections  between  the  load  bank  and  the 
batteries.  Circuit  breakers  protect  the  power  distribution  unit  from  dangerously  high  currents 
coming  from  the  batteries.  The  Power  Storage  unit  is  divided  into  two  major  subsystems:  the 
battery  cabinet  and  the  battery-load  selection  panel.  The  Power  Distribution  unit  consists  of  two 
identical  load  banks.  Each  load  bank  is  connected  to  the  Power  Storage  unit  and  powers  two  DC 
loads  and  six  AC  loads. 

A  complete  description  of  the  ADAPT  system  can  be  found  in  [1 1].  The  testbed  is 
controlled  by  a  number  of  relays  and  monitored  by  a  large  set  of  sensors.  Consequently,  it  is 
possible  to  detect  an  injected  fault  and  recover  from  it  if  the  correct  action  is  taken.  To  facilitate 
the  execution  of  the  experiments  performed  with  the  testbed,  three  operating  roles  have  been 
defined  [10];  user,  antagonist  and  observer.  The  user  simulates  an  actual  crewmember  or  pilot 
who  operates  and  maintains  the  EPS  with  the  help  of  a  vehicle  health  management  application. 
The  antagonist  injects  faults  into  the  system,  either  manually  by  physically  acting  on  the  system, 
or  remotely  by  spoofing  sensor  values  through  a  computer  connected  to  the  system.  The 
malicious  actions  of  the  antagonist  are  not  known  to  the  user  who  is  responsible  of  choosing  a 
suitable  recovery  action.  The  observer  logs  all  data  in  the  experiment  and  monitors  how  the  user 
responds  to  the  faults  injected  by  the  antagonist  and  therefore  measures  the  effectiveness  of  the 
test  article.  The  observer  also  acts  as  a  safety  officer  of  the  experiment  and  can  issue  an 
emergency  stop. 

4,1,2  Models 

The  Automated  Model  Generator  developed  in  the  framework  of  the  projects  translates  a 
feasible  EPS  architecture  generated  by  the  concept  generator  into  a  corresponding  model 
expressed  in  the  Modelica  Language. 

Modelica  is  a  language  for  hierarchical  object  oriented  physical  modeling,  which  is 
developed  through  an  international  effort.  The  language  unifies  and  generalizes  previous  object- 
oriented  modeling  languages  and  is  intended  to  become  a  de  facto  standard  for  modeling  and 
simulation  of  dynamical  systems.  The  language  has  been  designed  to  allow  tools  to  generate 
efficient  simulation  code  automatically  with  the  main  objective  to  facilitate  exchange  of  models, 
model  libraries  and  simulation  specifications.  It  allows  defining  simulation  models  modularly 
and  hierarchically  and  combining  various  formalisms  expressible  in  the  more  general  Modelica 
formalism.  The  multidomain  capability  of  Modelica  gives  the  user  the  possibility  to  combine 
electrical,  mechanical,  hydraulic,  thermodynamic  etc,  model  components  within  the  same 
application  model.  Modelica  is  primarily  a  modeling  language,  sometimes  called  hardware 
description  language,  that  allows  the  user  to  specify  mathematical  models  of  complex  physical 
systems,  e.g.  for  the  purpose  of  computer  simulation  of  dynamic  systems  where  behavior  evolves 
as  a  function  of  time.  Modelica  is  also  an  object-oriented  equation  based  programming  language, 

18 


Approved  for  public  release;  distribution  unlimited. 


oriented  towards  eomputational  applieations  with  high  eomplexity  requiring  high  performanee. 
The  four  most  important  features  of  Modeliea,  relevant  for  out  projeet,  are; 

•  Modeliea  is  primarily  based  on  equations  instead  of  assignment  statements.  This  permits 
aeausal  modeling  that  gives  better  reuse  of  elasses  sinee  equations  do  not  speeify  a  eertain 
data  flow  direetion.  Thus  a  Modeliea  elass  ean  adapt  to  more  than  one  data  flow  eontext.  This 
feature  of  the  Modeliea  language  was  extremely  useful  in  defining  eomplex  eomponent 
failure  mode  behavior  of  the  EPS  eomponents  that  gave  us  to  possibility  for  enhaneed  FFA 
and  PCC  analysis. 

•  Modeliea  has  multi-domain  modeling  eapability,  meaning  that  model  eomponents 
eorresponding  to  physieal  objeets  from  several  different  domains  sueh  as  e.g.  eleetrieal, 
meehanieal,  thermodynamie,  hydraulie,  biologieal  and  eontrol  applieations  ean  be  deseribed 
and  eonneeted.  The  EPS  eonsist  of  eomponents  from  the  eleetrieal  and  meehanieal  domain. 

•  Modeliea  is  an  objeet-oriented  language  with  a  general  elass  eoneept  that  unifies  elasses, 
generies  —  known  as  templates  in  C++,  and  general  subtyping  into  a  single  language 
eonstruet.  This  faeilitates  reuse  of  eomponents  and  evolution  of  models. 

•  Modeliea  has  a  strong  software  eomponent  model,  with  eonstruets  for  ereating  and 
eonneeting  eomponents.  Thus  the  language  is  ideally  suited  as  an  arehiteetural  deseription 
language  for  eomplex  physieal  systems,  and  to  some  extent  for  software  systems. 

Modeliea  defines  a  standard  mapping  of  hierarehieal  paekage  struetures  onto  file  systems 
or  other  storage  meehanisms  sueh  as  databases,  whieh  provides  the  user  with  a  simple  and 
unambiguous  way  of  loeating  a  paekage.  The  Modeliea  library  path  meehanism  makes  it  possible 
to  make  multiple  paekages  and  paekage  hierarehies  simultaneously  available  for  lookup.  All 
these  meehanisms  give  the  user  a  eonsiderable  flexibility  and  power  in  strueturing  a  paekage. 
Modeliea  defines  a  method  for  loeating  a  paekage  by  providing  a  standard  mapping  of  paekage 
names  to  storage  plaees,  typieally  file  or  direetory  loeations  in  a  file  system.  The  Modeliea 
format  gave  us  the  possibility  to  easily  store  the  model  eomponents  in  eomponent  library 
repositories  that  ean  be  easily  aeeessed  and  retrieved  for  automatieally  building  feasible  EPS 
arehiteetures.  Figure  8  shows  the  strueture  of  the  EPS  Modeliea  performanee  library  and  Figure 
9  shows  the  graphieal  representation  of  the  EPS  library  model  eomponents. 
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Figure  8.  The  EPS  Modelica  Performance  Components  Library  Structure 


Given  an  input  XML  concept  file  representing  the  Automated  Model  Generator  is  able  to 
build  following  models: 
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•  Modelica  EPS  Performance  models  for  simulation  purposes,  performance  analysis 
uncertainty  propagation  and  PCC  Computation. 

•  Modelica  EPS  Reliability  models  for  performing  reliability  analysis  computations  as 
described  in  Section  3.9. 

•  Modelica  EPS  Eunctional  models  for  performing  Eunctional  Eailure  Analysis  as  described  in 
Section  3.5. 

Eigure  10  illustrates  the  types  of  models  generated  by  the  Automatic  Model  Generator. 


Eigure  10;  Type  of  Modelica  models  generated  by  the  Automated  Model  Generator 


Eigures  1 1  and  12  depict  two  feasible  EPS  architectures  automatically  generated  from  the 
concept  generator. 
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Figure  11.  Automatically  generated  Modelica  performance  with  redundant  power 

sources. 


Figure  12.  Automatically  generated  Modelica  performance  model  with  one  power  source 

and  multiple  loads. 


4,1.3  Findings 

For  this  case  study,  uncertainty  is  assumed  in  two  system  parameters:  1)  inverter  output 
voltage  and  2)  fan  motor  resistance.  For  the  three  design  phases,  concept  generation,  detailed 
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design  and  verification,  the  same  Modelica  model  is  used;  however  the  representation  of 
uncertainty  is  varied  as  follows: 

Phase  1 :  In  this  phase,  both  parameter  uncertainties  are  assumed  normally  distributed: 
Inverter  output  voltage  ~N(  120.0,  4.0)  and  fan  motor  resistance  ~N(  131.8,  2.2)  .  The  use  of  the 
normal  approximation  of  uncertainty  represents  early  design,  in  which  general  tolerances  for  a 
given  component  type  would  be  known,  but  not  the  actual  performance  distribution. 

Phase  2:  In  this  case,  the  inverter  output,  which  is  specified  by  the  manufacturer  to  have 
output  of  120V AC  +  3%  /  -10%,  is  modeled  using  a  four  parameter  beta  distribution,  Beta(  4.0, 
1.9,  108.0,  123.6)  ,  to  provide  a  distribution  with  +  3%  /-  10%  range  and  mode  of  120.  The  fan 
resistance  is  modeled  as  a  uniform  distribution,  U(  125.2,  138.4)  .  This  case  more  accurately 
represents  actual  engineering  uncertainty  in  the  parameters,  which  would  be  known  in  the 
detailed  design  phase. 

Phase  3:  In  this  case,  the  inverter  output  and  fan  resistance  have  the  same  distributions  as 
Phase  2;  however,  a  full  MCS  is  conducted  to  verify  the  PoC  computations  of  Phase  2. 

These  uncertainties  are  propagated  through  the  ADAPT  Modelica  system  model  using 
the  six  methods  described  in  Section  3.8.  For  each  phase,  all  six  uncertainty  methods  are  utilized 
for  illustrative  purposes  to  compare  accuracy.  A  single  requirement  is  considered  in  the 
simulation:  “fan  speed  to  be  greater  than  or  equal  to  780  RPM”,  The  results  of  the  study  are 
shown  in  Table  1  for  Phase  1  (recommended  Phase  1  methods  in  bold)  and  Table  2  for  Phases  2 
&  3  (recommended  Phase  2  &  3  methods  in  bold). 


Table  1.  Results  for  the  ADAPT  EPS: 
Design  Phase  1 . 


Method 

TS 

MPP 

UDR 

FFNI 

PCE 

MCS 

mean 

868.2405 

* 

868.1354 

868.1347 

868.1347 

868.0543 

stddev 

38.3735 

-* 

38.2952 

38.2866 

38.2866 

38.1584 

skewness 

* 

-* 

0.0300 

0.0220 

0.0220 

0.0180 

kunosis 

♦ 

* 

3.0000 

29970 

3.0010 

3.0535 

dist 

M 

M 

Beta 

Beta 

PearlV 

* 

PoC 

0.‘)8‘)3 

0.0800 

0.9887 

0.9889 

0.9889 

0.9890 

#  F  calls 

5 

16 

7 

10 

27+10 

10.000 

Table  2.  Results  for  the  ADAPT  EPS: 
Design  Phase  2  and  3. 


Method 

TS 

MPP 

UDR 

FFNI 

PCE 

MCS 

mean 

854.9603 

* 

855.1931 

855.1921 

855.1844 

855.3800 

std  dev 

35.3258 

* 

30.4884 

30.4747 

30.3623 

30.5190 

skewness 

* 

* 

OJ130 

0.2890 

0.2980 

0.2857 

kunosis 

* 

* 

2.7450 

2.7210 

2.7460 

2.7048 

dist 

M 

Beta 

Beta 

Beta 

* 

PoC 

0.9831 

0.9866 

0.9901 

0.9907 

0.9906 

0.9914 

#  F  calls 

5 

20 

7 

10 

125+26 

10.000 

23 


Approved  for  public  release;  distribution  unlimited. 


The  tables  inelude  the  first  four  moments  (mean,  std  dev,  skewness,  and  kurtosis),  the 
distribution  of  the  output,  the  PoC,  and  the  number  of  funetion  ealls  for  eomparison.  In  the  ease 
of  the  UDR  and  FFNI  methods,  a  3-node  approximation  is  used  for  both  Cases  1  &  2.  For  the 
Phase  1  PCE,  a  seeond  order  (i.e.,  p  =  2)  polynomial  approximation  with  3-node  quadrature  for 
estimating  the  moments  is  used.  For  the  Phase  2  &  3  PCE,  a  fourth  order  (i.e.,  p  =  4) 
approximation  with  5 -node  quadrature  is  required  to  aeeurately  estimate  the  moments  due  to  the 
non-normality  of  the  input  and  output  distribution,  for  Phase  2  &  3,  the  inputs  are  non-normal 
and  the  Rosenblatt  transformation  is  used  for  the  MPP  and  PCE  requiring  normal  inputs,  for  the 
TS  method,  the  mean  and  standard  deviation  of  the  two  input  distributions  is  used  direetly; 
however,  the  output  is  assumed  to  be  normal,  for  the  UDR  and  f FNI  methods,  quadrature 
methods  are  available  to  provide  nodes  and  weights  for  the  beta  and  uniform  distributions 
direetly  without  transformation  to  normal  spaee.  The  MCS  method  can  also  handle  non-normal 
inputs  directly  without  transformation. 

The  output  distribution  resulting  from  the  MCS  for  Phase  1  is  shown  in  figure  13.  This 
output  distribution  is  approximately  normal  as  seen  by  the  shape  of  the  distribution  and  the  fact 
that  the  skewness  ~0  and  kurtosis  ~3.  Despite  the  fact  that  the  output  is  close  to  normal,  the 
Pearson  distribution  system  fits  alternative  distributions  to  the  output  based  on  the  deviations  of 
the  skewness  and  kurtosis  from  the  normal  distribution  as  shown  in  Table  2.  All  methods  provide 
good  approximations  of  the  PoC  (using  the  MCS  as  the  baseline)  for  Phase  1 .  This  is  because  the 
inputs  are  normal  and  the  system  model  is  not  highly  non-linear.  The  assumption  that  inputs  are 
normal  and  the  model  is  not  highly  non-linear  is  appropriate  for  the  concept  design  phase  when 
models  are  of  low  fidelity  and  uncertainty  distributions  are  approximated.  The  output  distribution 
for  Phase  2  &  3  resulting  from  the  MCS  is  shown  in  figure  14.  As  seen  by  the  skewness  and 
kurtosis  in  Table  2  and  the  shape  of  the  output  distribution,  this  output  deviates  considerably 
from  a  normal  distribution,  exhibiting  significant  skewness  and  lower  kurtosis  than  the  normal 
distribution.  As  shown  in  the  table,  there  are  greater  differences  in  the  PoC  estimates  in  Phase  2 
and  3  than  in  Phase  1 .  One  reason  for  the  deviation  is  that  methods  which  assume  the  output  is 
normal,  i.e.,  TS  and  MPP,  do  not  account  for  the  skewness  and  kurtosis  of  the  the  actual  output 
distribution.  The  TS  method  also  suffers  from  the  fact  that  only  the  first  two  moments  of  the 
input  distribution  are  considered,  and  the  distribution  of  inverter  voltage  is  significantly  skewed. 
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Figure  13.  Phase  1  Output  Distribution  from  Figure  14.  Phase  2/3  Output  Distribution  from 
MCS.  MCS. 


4,2  IFV  Ramp  Example 

4,2,1  Domain  Description 

To  demonstrate  the  features  of  the  proposed  verification  process  and  the  scalability  of  the 
methodology  and  the  tools  proposed  by  the  META  11  project,  a  Ramp  System  of  an  Infantry 
Fighting  Vehicle  (IFV)  has  been  modeled.  A  power-operated  ramp  at  the  rear  of  the  vehicle  is 
used  for  fast  exit  and  entry  of  troops  and  the  ramp  is  also  fitted  with  a  door  (Figure  15  illustrates 
the  rear  ramp  of  a  Bradley  Fighting  Vehicle). 


Figure  15.  A  BAE  Bradley  Fighting  Vehicle  (Courtesy  of  BAE  Systems) 
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Several  structural,  performance  and  safety  requirements  have  been  preliminary  imposed  on  the 
design  and  functionality  of  the  rear  ramp  system.  Table  3  below  illustrates  some  of  the 
performance  requirements  for  the  rear  ramp  system. 

Table  3.  Performance  Requirements  for  the  Rear  Ramp  System 


Performance  Requirements 

1. 

The  ramp  shall  accommodate  a  static  weight  of  320  kg  when  in  fully  open  position. 

2. 

The  ramp  shall  accommodate  a  static  weight  of  320  kg  when  in  partially  open  position  with  no  more  than  1  cm  of  play  in  either  direction. 

3. 

The  ramp  shall  operate  while  the  CFV  is  inclined  upslope  at  an  angle  of  60%  {27  degrees). 

4. 

The  ramp  shall  operate  while  the  CFV  is  inclined  downslope  at  an  angle  of  60%  (27  degrees). 

5. 

The  ramp  shall  operate  while  the  CFV  is  on  a  level  surface. 

6. 

With  the  engine  running,  the  ramp  shall  operate  from  the  fully  closed  position  to  the  fully  open  position  in  10  seconds  or  less. 

7. 

With  the  engine  running,  the  ramp  shall  operate  from  the  fully  closed  position  to  the  fully  open  position  in  10  seconds  or  less. 

4.2.2  Models 

A  hierarchical  Modelica  model  (see  Figure  16)  of  the  Ramp  System  has  been  built.  This 
model  consists  of  an  electrical  power  subsystem  (Figure  17),  a  control  subsystem  (Figure  18),  a 
mechanical  subsystem  (Figure  19),  and  a  crew  subsystem  (Figure  20). 


Figure  16.  The  Modelica  Model  of  a  Rear  Ramp  System 
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Figure  19.  The  Mechanical  Subsystem 


Figure  20.  The  Crew  Subsystem 


4,2.3  Findings 

Uncertainty  was  induced  in  15  components  as  shown  in  Table  4.  The  tolerance  limits 
were  used  to  model  uncertainty  in  the  inputs  using  4  parameter  beta  distributions.  The 
uncertainty  in  the  components  creates  uncertainty  in  meeting  the  system  requirements. 
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Table  4.  Model  Stoehastie  Inputs 


Stochastic  Inputs 

Uncertainty 

EPS  Subsystem 

1 

Fan.rNom 

141ohms  +/-S% 

2 

Pump.R 

lohm  +I-S% 

3 

Battery. V 

24V  +/-7% 

4 

Inverterl.outputvoltage 

120V  +4%  -10% 

Combat  Crew  Subsystem 

5 

SoldierOl. duration 

2  +/-10% 

6 

Soldier02. duration 

1.5+/- 10% 

7 

Soldier03. duration 

3  +/-10% 

8 

Soldier04. duration 

2  +/-10% 

9 

SoldierOS. duration 

1  +/-10% 

10 

SoldierOl. height 

1000  +/-20% 

11 

Soldier02. height 

1000  +/-20% 

12 

Soldier03. height 

1000  +/-20% 

13 

Soldier04. height 

1000  +/-20% 

14 

SoldierOS. height 

1000  +/-20% 

Gravity  load  Subsystem 

15  yehicle_angle.k 

0  +/-  0.54 

As  shown  in  Table  5,  nine  representative  system  outputs  corresponding  to  system 
requirements  were  monitored,  and  the  probability  of  meeting  each  of  these  requirements  was 
computed  (note  that  some  requirements  are  listed  as  meeting  the  requirement  with  Pr=1.0  when 
in  actuality  the  probability  approaches  but  never  actually  reaches  1.0).  In  addition  to  computing 
the  probability  of  meeting  each  of  the  nine  requirements,  the  joint  probability  of  meeting  all  the 
requirements  is  also  computed,  which  is  Pr=0.7229.  This  joint  probability  is  enabled  by 
computation  of  the  variance-covariance  matrix  for  the  set  of  requirements. 
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Table  5.  Probability  of  Correctness  for  the  Requirements 


Requirement 

PoC 

Fan  Flow  Rate 

0.8603 

Input  Voltage 

0.9798 

Light  Temperature 

1.0000 

Speed  Deviation 

0.9970 

Door  Angle  with  Load 

0.8614 

Controller  torque  (internal) 

1.0000 

Gravity  torque  (internal) 

1.0000 

Solder  torque  (internal) 

1.0000 

Opening  Time  @  -60Deg 

1.0000 

Joint  PoC 

0.7229 

An  example  an  output  distribution  is  shown  in  Figure  21,  which  is  the  distribution  of  the 
deviation  of  the  actual  ramp  speed  from  the  reference  signal.  As  seen  in  this  distribution,  the 
requirement  is  that  the  speed  deviation  be  less  than  0.015  and  due  to  the  variation  in  speed 
caused  by  component  uncertainties,  there  is  some  area  in  the  upper  tail  region  which  is  beyond 
the  requirement  limit,  resulting  in  a  probability  of  meeting  the  requirement  of  0.997. 


Figure  21.  Example  of  Speed  Deviation  Requirement 


Calculation  of  the  individual  probabilities  and  the  joint  probability  of  meeting  a  set  of 
requirements  enable  the  designer  to  target  where  design  refinement  is  needed  to  better  meet 

30 


Approved  for  public  release;  distribution  unlimited. 


requirements.  To  understand  which  components  are  driving  uncertainties  in  meeting 
requirements,  a  hierarchical  sensitivity  analysis  (SA)  is  conducted  as  shown  in  Figure  22.  At  the 
system  level,  the  SA  shows  that  the  Combat  Vehicle  Crew  accounts  for  most  of  the  variation  in 
the  Ramp  Speed  (70.06%)  and  the  Ramp  Angle  (89.49%).  Next  the  Combat  Vehicle  Crew 
subsystem  is  analyzed  to  determine  the  drivers  of  variation  in  this  subsystem.  The  results  show 
that  most  of  the  variation  is  caused  by  the  solder  step  height  2  (48.1 1%)  and  solder  step  height  3 
(44.66%).  This  indicates  to  the  designer  than  more  focus  must  be  placed  on  the  solder  step 
heights  in  the  design  to  reduce  variation  in  the  system  outputs  to  increase  the  probability  of 
meeting  requirements. 


Figure  22.  Hierarchical  Sensitivity  Analysis 
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5.0  CONCLUSIONS 


In  this  project,  we  have  developed  a  framework  and  tool  chain  for  model-based  system 
engineering  and  design  that  integrates  architectural  synthesis  and  analysis  of  complex  systems 
during  the  conceptual  design  phase.  The  incorporation  of  automated  design  space  exploration 
methods  with  detailed  functional  and  behavioral  models  enables  architectural  trade  studies 
before  costly  design  decisions  are  made.  In  future  efforts,  we  plan  to  fully  integrate  and  automate 
the  architectural  synthesis  and  analysis  approaches  described  in  this  report.  We  are  also  planning 
to  improve  the  coverage  of  requirements  that  are  addressable  using  our  approach  and  to  measure 
the  performance  of  our  tools  in  comparison  to  human  teams  performing  design  tasks  of  similar 
complexity. 
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APPENDIX  A:  PROJECT  TEAM 


Palo  Alto  Research  Center  (PARC) 

Serdar  Uckun 

Serdar  Uckun  is  the  Principal  Investigator  for  the  PARC  META-II  projeet.  He  is  a  Principal 
Scientist  at  PARC.  Prior  to  PARC,  he  was  at  NASA  Ames  Researeh  Center  where  he  led  the 
largest  organization  in  the  government  focusing  on  prognostics  and  health  management  (PHM) 
researeh.  Earlier,  he  served  as  Direetor  of  the  Researeh  Institute  for  Advaneed  Computer 
Seienee  (RIACS),  Direetor  of  Advaneed  Teehnology  at  Blue  Pumpkin  Software,  and  Assistant 
Direetor  of  Roekwell  Seienee  Center  -  Palo  Alto  Eaboratory.  Dr.  Uckun  has  graduate  degrees  in 
Medieine  and  Biomedieal  Engineering,  and  he  has  eompleted  post-doctoral  studies  in  Computer 
Seienee  at  Stanford.  His  teehnieal  interests  inelude  diagnosis,  prognosties,  and  optimization.  He 
served  as  Assoeiate  Editor  of  the  Artifieial  Intelligenee  in  Medieine  Journal  and  the  General 
Chair  of  the  2008  International  Conferenee  on  PHM.  He  is  an  Assoeiate  Editor  of  the 
International  Journal  on  PHM.  Additionally,  he  is  founder  and  President  of  the  PHM  Soeiety,  a 
non-profit  professional  organization.  He  holds  fourteen  U.S.  patents. 

Tolga  Kurtoglu 

Dr.  Tolga  Kurtoglu  is  Prineipal  Investigator  for  the  PARC  META-II  projeet  and  area  manager  of 
the  Automation  for  Engineered  Systems  (AES)  group  at  PARC.  His  researeh  focuses  on  the 
design  and  development  of  eomplex  systems,  design  theory  and  methodology  with  a 
specialization  in  conceptual  design,  design  automation  and  optimization,  and  artificial 
intelligence  in  design.  He  eonduets  researeh  in  the  areas  of  development  of  prognostie  and  health 
management  teehnologies,  model-based  diagnosis,  automated  reasoning,  systems  engineering, 
and  risk  and  reliability-based  design.  Dr.  Kurtoglu  has  published  over  50  artieles  and  papers  in 
various  journals  and  eonferenees  and  is  an  aetive  member  of  ASME,  AIAA,  AAAI,  ASEE, 
Design  Soeiety,  and  the  Prognostics  and  Health  Management  Soeiety.  Prior  to  his  work  with 
PARC,  he  worked  as  a  researcher  at  NASA  Ames  Researeh  Center  and  as  a  systems  design 
engineer  and  lead  at  Dell  Corporation. 

Christian  Fritz 

Dr.  Christian  Eritz  is  a  researeh  seientist  at  the  Intelligent  Systems  Eaboratory  of  PARC.  His 
researeh  interests  inelude  logie-based  knowledge  representation,  symbolie  reasoning,  planning 
with  preferenees,  exeeution  monitoring,  and  workflow  synthesis.  Before  joining  PARC,  Dr.  Eritz 
did  postdoetoral  studies  at  the  Information  Scienees  Institute.  Dr.  Eritz  reeeived  his  Ph.D.  in 
eomputer  seienee  from  the  University  of  Toronto,  Canada,  in  2009,  for  whieh  he  reeeived 
honorable  mention  of  the  Best  Dissertation  Award  of  the  International  Conferenees  on 
Automated  Planning  and  Seheduling  (I CAPS)  in  2010.  He  also  holds  a  baehelor's  and  master's 
degree  in  eomputer  seienee  from  the  RWTH  Aaehen  University,  Germany  (2003).  Eor  his 
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Master’s  thesis,  he  was  awarded  the  Springorum-Denkmuenze  (University  Medal)  at  RWTH 
Aachen  in  2004.  In  2007,  Dr.  Fritz  received  the  Best  Paper  award  at  the  ICAPS  Doctoral 
Consortium.  Dr.  Fritz  regularly  serves  as  reviewer  for  prestigious  journals  in  the  area  of  artificial 
intelligence  and  on  the  (senior-)  program  committees  of  all  of  the  top  conferences  in  his  field. 

Peter  Bunus 

Dr.  Peter  Bunus'  primary  interest  is  in  investigating  model-based  techniques  for  system 
engineering.  His  research  philosophy  involves  tackling  practical  problems  that  are  relevant  and 
important  to  the  current  issues  in  model-based  systems  research  and  propose  foundational 
solutions  to  them  for  good  impact.  Currently,  his  research  is  focused  on  modeling  techniques, 
modeling  language  development,  and  model-driven  engineering  with  direct  applications  to 
diagnostics  and  automated  program  and  model  verification.  Peter  has  vast  experience  in 
conducting  research  activities  with  the  ability  to  bring  research  ideas  into  production  and 
development.  He  enjoys  acting  as  liaison  between  sales,  marketing,  and  technology  groups  to 
support  product  positioning  and  customer  demand  as  part  of  new  product  development.  Prior  to 
joining  PARC,  Peter  was  involved  in  several  start-up  high  technology  ventures  as  a  co-founder 
and  has  been  actively  involved  in  building  their  businesses.  Peter  also  holds  an  Associate 
Professor  position  in  Computer  Science  at  Linkoping  University  Sweden  where  he  is  teaching 
Computer  Science  and  Software  Engineering  courses  at  graduate  and  undergraduate  level,  and 
performs  research  in  the  area  of  model-driven  engineering. 

Peter  Jarvis 

Peter  is  a  seasoned  engineer  and  scrum  master  with  a  track  record  of  delivering  quality  software 
products  on  time  and  budget.  He  came  to  PARC  in  December  2010  after  a  seven  year  stint  at 
NASA  where  he  developed  the  next  generation  plotting  software  for  shuttle  and  space  station 
flight  controllers.  Prior  to  NASA,  Peter  spent  four  years  at  SRI  International  working  on  DARPA 
and  ARDA  programs.  He  holds  a  PhD  in  Artificial  Intelligence  from  The  University  of  Brighton 
(UK)  and  has  over  twenty  publications  in  international  conferences. 


Oregon  State  University 
Irem  Turner 

Dr.  Irem  Y.  Turner  is  an  Associate  Professor  at  Oregon  State  University  (OSU),  where  she  leads 
the  Complex  Engineered  System  Design  Eaboratory.  Her  research  focuses  on  the  overall 
problem  of  designing  highly  complex  and  integrated  engineering  systems  with  reduced  risk  of 
failures,  and  developing  formal  methodologies  and  approaches  for  complex  system  design  and 
analysis.  Her  expertise  touches  on  topics  such  as  risk-based  design,  systems  engineering, 
function-based  design,  failure  analysis,  and  model-based  design.  Since  moving  to  OSU  in  2006, 
her  funding  has  largely  been  through  NSE,  AEOSR,  DARPA,  and  NASA.  Prior  to  accepting  a 
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faculty  position  at  OSU,  Dr.  Turner  led  the  Complex  Systems  Design  and  Engineering  group  in 
the  Intelligent  Systems  Division  at  NASA  Ames  Researeh  Center,  where  she  worked  from  1998 
through  2006  as  Research  Scientist,  Group  Lead,  and  Program  Manager.  She  reeeived  her  Ph.D. 
in  Mechanical  Engineering  from  The  University  of  Texas  at  Austin  in  1998. 

Chris  Hoyle 

Dr.  Christopher  Hoyle  is  eurrently  an  Assistant  Professor  in  the  area  of  Design  in  the  Mechanical 
Engineering  Department  at  Oregon  State  University.  His  current  researeh  interests  are  foeused 
upon  deeision  making  in  engineering  design,  with  emphasis  on  the  early  design  phase  when 
uncertainty  is  high  and  the  potential  design  space  is  large.  His  research  contributions  are  to  the 
field  of  Decision-based  Design,  specifically  in  linking  consumer  preferences  and  enterprise-level 
objectives  with  the  engineering  design  proeess.  His  areas  of  expertise  are  uneertainty 
propagation  methodologies,  Bayesian  statistics  and  modeling,  stochastie  consumer  choice 
modeling,  optimization  and  design  automation.  He  is  currently  coauthoring  the  book  Deeision- 
Based  Design:  Integrating  Consumer  Preferences  into  Engineering  Design,  to  be  published  in 
2012.  He  reeeived  his  PhD  from  Northwestern  University  in  Mechanical  Engineering  in  2009 
and  his  Master’s  degree  in  Mechanical  Engineering  from  Purdue  University  in  1994.  He  served 
as  an  Adjunct  Professor  of  Mechanical  Engineering  at  Illinois  Institute  of  Teehnology  in  2009 
and  was  a  Summer  Intern  at  the  National  Aeronautics  and  Space  Administration  (NASA)  Ames 
Researeh  Center  in  2006.  He  was  previously  a  Design  Engineer,  an  Engineering  Manager,  and  a 
Program  Manager  at  Motorola,  Ine.  for  10  years  before  enrolling  in  the  PhD  program  at 
Northwestern  University. 

David  Jensen 

David  Jensen  is  a  PhD  student  and  research  assistant  at  Oregon  State  University.  He  earned  both 
a  Baehelor  and  Master  of  Seienee  from  Oregon  State  University  in  Meehanical  Engineering  in 
2008  and  2009  respectively.  His  researeh  to  date  has  focused  on  funetion-based  failure 
simulation  and  analysis.  He  has  worked  as  a  summer  intern  in  Intelligent  System  Division  at 
NASA  Ames  Researeh  Center  and  as  a  researeher  at  Aalto  University  in  Helsinki,  Einland. 
Previous  work  in  eomplex  software -hardware  produet  design  and  testing  has  provided  him  with  a 
baekground  and  interest  in  failure  identification  and  mitigation.  His  research  has  been  published 
in  the  ASME  International  Meohanieal  Engineering  Congress  and  Expo  and  International  Design 
Engineering  Teehnieal  Conferences,  the  IEEE  Aerospace  Conference  and  Prognostics  and 
Health  Management  Soeiety  Conferenee. 


Smart  Information  Flow  Technologies  (SIFT) 

Dave  Musliner 

Eor  his  Ph.D.  dissertation.  Dr.  Musliner  designed  and  implemented  the  Cooperative  Intelligent 
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Real-Time  Control  Arehiteeture  (CIRCA),  one  of  the  first  AI  planning  and  eontrol  arehiteetures 
eapable  of  reasoning  about  and  interaeting  with  dynamie,  hard  real-time  domains.  CIRCA  is 
designed  to  make  rigorous  safety  guarantees  about  its  plans,  using  formal  verifieation  methods  to 
ensure  that  its  plans  (reactive  controllers)  avoid  predicted  failures  while  also  achieving  system 
goals. 

In  collaboration  with  other  current  SIFT  researchers.  Dr.  Musliner  incorporated  formal  model 
checking  methods  into  CIRCA.  Taking  advantage  of  the  unique  aspects  of  CIRCA's  real-time 
plans,  this  research  team  developed  a  novel  model  checking  algorithm  that  improved  system 
scalability  by  two  orders  of  magnitude  and  led  to  a  U.S.  patent  on  incremental  verification.  Dr. 
Musliner  also  led  the  extension  of  CIRCA  to  reasoning  about  probabilistic  systems,  developing 
the  first  statistical  model  checking  systems  for  use  in  verifying  CIRCA's  automatically-designed 
controllers.  In  addition  to  research  work  on  CIRCA,  Dr.  Musliner  has  applied  intelligent  control 
and  model  checking  verification  technologies  to  various  real-world  industrial  problems, 
including  advanced  control  of  oil  refineries  and  aircraft  software. 

Eric  Engstrom 

Eric  Engstrom  has  a  broad  expertise  in  software  engineering  and  specific  interests  in  the  areas  of 
model  based  development  and  formal  methods.  During  his  17  years  at  Honeywell’s  corporate 
research  center,  one  of  Mr.  Engstrom’ s  major  contributions  is  the  creation  of  DOME  (Domain 
Modeling  Environment),  as  highly-regarded  open  source  meta-modeling  research  platform 
providing  a  user-extensible  model-based  development  environment.  In  collaboration  with  other 
NASA  and  Honeywell  researchers,  Mr.  Engstrom  used  the  explicit-state  model  checking  tool 
SPIN  and  the  theorem  proving  tool  PVS  to  support  automated  verification  of  time  partitioning  in 
the  Honeywell  DEOS  real-time  avionics  scheduling  kernel. 


Mission  Critical  Technologies 
Jacquelyn  Nagel 

Jacquelyn  K.  Nagel,  Ph.D.  is  an  engineering  contractor  at  Mission  Critical  Technologies  working 
on  engineering  design  focused  projects,  and  has  six  years  of  diversified  engineering  design 
experience,  both  in  academia  and  industry.  Dr.  Nagel  has  experienced  engineering  design  in  a 
range  of  contexts,  including:  product  design,  biomimetic  design,  electrical  and  control  system 
design,  manufacturing  system  design  and  design  for  the  factory  floor.  She  earned  her  Ph.D.  in 
Mechanical  Engineering  from  Oregon  State  University,  and  her  M.S.  and  B.S.  in  Manufacturing 
Engineering  and  Electrical  Engineering,  respectively,  from  the  Missouri  University  of  Science 
and  Technology  (formerly  known  as  University  of  Missouri-Rolla).  As  a  student,  she  worked  at 
Kimberly-Clark,  Motoman,  and  Intel  and  gained  cooperative  education  experience  in  the  areas  of 
industrial  automation,  manufacturing,  and  design.  Dr.  Nagel’s  long-term  goal  is  to  drive 
engineering  innovation  by  applying  her  multidisciplinary  engineering  expertise  to  design, 
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analysis,  instrumentation,  and  manufacturing  challenges. 


David  Kluck 

Dave  Kluck  graduated  from  Cal  Poly,  San  Luis  Obispo  with  a  B.S.  degree  in  Computer 
Science.  He  has  worked  as  a  software  developer  for  the  past  25  years.  Dave  started  out  in  the 
image  processing  and  analysis  field  writing  a  product  line  of  software  used  by  various 
universities  and  medical  institutions.  For  the  past  16  years,  he  has  worked  for  Mission  Critical 
Technologies,  Inc.  designing  and  developing  a  multitude  of  systems  using  an  ever-increasing 
array  of  technologies.  These  technologies  have  included,  but  are  not  limited  to,  the  more 
traditional  languages  of  assembly,  BASIC,  FORTRAN,  COBOL,  C,  and  C++  to  the  web- 
oriented  languages  and  frameworks  of  PHP,  Python,  Django,  ASP,  HTML,  and  Javascript.  Dave 
also  has  extensive  knowledge  and  experience  working  with  a  variety  of  databases  including,  but 
not  limited  to,  DB2,  Oracle,  SQL  Server,  and  MySQL.  Dave  has  designed  and  developed 
systems  for  DOS,  Windows,  and  UNIX  ranging  from  sales  tracking,  reporting,  and  analysis  to  e- 
commerce  solutions  to  e-leaming  to  website  design  and  maintenance  to  Web  2.0 
applications.  These  systems  have  been  designed  for  a  wide  array  of  businesses  including,  but  not 
limited  to,  the  entertainment  industry,  food  services  industry,  aerospace  industry,  medical 
industry,  and  the  government. 


Carnegie  Mellon  University 
Matt  Knudson 

Dr.  Matt  D.  Knudson  is  currently  a  research  scientist  at  the  NASA  Ames  Research  Center  with 
Carnegie  Mellon  University.  His  research  interests  are  in  autonomous  and  adaptable  systems, 
particularly  robotics  and  complex  multiagent  organizations.  These  include  low-resource 
intelligent  decision  making  processes  for  robot  navigation  and  coordination  methodologies  for 
large  heterogeneous  multiagent  systems,  such  as  teams  of  exploration  robots  and  distributed 
sensor  networks.  His  expertise  is  in  the  fields  of  electronics  and  computer  science,  particularly 
artificial  intelligence.  Specifically,  he  develops  representations  for  system  states  and  actions,  as 
well  as  algorithms  that  operate  on  such  representations,  for  agents  to  make  intelligent  decisions 
with  as  little  information  as  possible.  In  addition,  he  designs  objective  functions  for  agents  that 
promote  inherent  coordination,  without  the  need  for  communication.  Prior  to  accepting  a  position 
at  NASA  Ames,  he  worked  as  a  post-doctoral  researcher  at  Oregon  State  University,  where  he 
was  the  supervisor  of  the  Autonomous  Agents  and  Distributed  Intelligence  laboratory.  During 
this  time,  he  supervised  or  assisted  in  the  research  of  graduate  students  working  under  NSF, 
AFOSR,  and  DoE  funding. 
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LIST  OF  ACRONYMS  AND  ABBREVIATIONS 

ACRONYM 

DESCRIPTION 

ADAPT 

Advanced  Diagnostics  and  Prognostics  Testbed 

DARPA 

Defense  Advanced  Research  Projects  Agency 

DDTE 

Design,  development,  test,  and  evaluation 

EPS 

Electrical  Power  System 

FFA 

Functional  Failure  Analysis 

FMEA 

Failure  Modes  and  Effects  Analysis 

FMECA 

Failure  Modes,  Effects,  and  Criticality  Analysis 

IC 

Integrated  circuit 

PARC 

Palo  Alto  Research  Center 

PCC 

Probabilistic  Certificate  of  Correctness 

PoC 

Probability  of  Correctness 

SA 

Sensitivity  Analysis 

UP 

Uncertainty  Propagation 
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